Parallel fitting #17
Merged
Parallel fitting #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…longest time within _pyse, so we want to parallellize the fitting loop over all the CPU cores. It does not make sense to distribute all islands over threads separately, because it will impose too much overhead. We encountered this before when kappa, sigma clipping subimages: rows of subimages turned out to be preferable. Now we are doing something similar as rows of subimages: island_block, i.e. a list of islands of a size convenient for a single CPU core. This commit is an intermediate step and does not invoke parallellisation yet.
…ction will get a dask.delayed and staticmethod decorator and then it should work. Now it is still an instance method, so some refactoring will be needed.
…tps://stackoverflow.com/questions/735975/static-methods-in-python/738102#738102 Also, the variable name chunk_size was used elsewhere, so I renamed it.
…actually five times slower!
…mewhat "cleaner". However, the fitting time has increased and the resulting island map is empty.
…and in island_list:' loop in ImageData._pyse. It seemed hard to construct a static or class method for e.g. dask.delayed or dask.bag since there were no many dependencies on the instance. Then I decided to try to parallellize the first part of this loop - i.e. fit_results = island.fit(fixed=fixed), no instance dependencies - only and leave the rest as serial code. This worked out really well, because the rest of the 'for island in island_list:' turned out to take a short time. This parallellisation of the fitting loop should conclude all pyse parallellisations. Any significant additional speedups can only come from translating Python code to e.g. C, I reckon.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This parallellisation of the fitting of sources should be the last part of the parallellisation of pyse.
This does not mean that it is as fast as it could be.
That would require translation of some routines to C-libraries.
But I think all the low-hanging fruit wrt straightforward parallellisation has been picked with this last merge.