-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix mpl and multithreading #84
Conversation
Kojobu
commented
Jun 25, 2024
- Use modern matplotlib approach to register colormaps
- Fix multiprocessing (might be an issue with local hardware, please test before merging this!)
2) fix mulitprocessing, Pool by adding mp_context
2) fix mulitprocessing, Pool by adding mp_context
Thanks @Kojobu for the PR. I think the modern MPL part is a 21cmFAST thing, not 21cmMC -- there's actually an open PR there that fixes it. The |
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
The multiprocessing seems to be a local problem, you may skip this. The small fix for matplotlib may be worth a look though. |
@Kojobu OK thanks for noting that. The fix for matplotlib is not actually in this PR (see the changed files tab above). The problem is in 21cmFAST. |
@steven-murray You are right, my bad. So, I'm on Linux, Python version 3.11 and I consistently get memory leaks when running with INHOMO_RECO and TS_FLUCT. After too many trial and errors I found it's some issue with Python's concurrent.futures.ProcessPoolExecutor. set_start_method('spawn')
pool = mcmc_options.pop(
"pool",
ProcessPoolExecutor(
max_workers=mcmc_options.get("threadCount", 1),
max_tasks_per_child=1,
),
) What this does is, it explicitly uses the spawn method (fork, which is standard in Linux, will be removed in Python3.14 according to the docs anyway) which creates a new process every run. It's kind of a brute force approach admittedly but performance-wise it shouldn't be much slower and it works. |