Replies: 1 comment 5 replies
-
|
Hi, I'm glad to hear you find future useful. It's really hard to say something useful without even a code example. One clue is in the following part of the error: That means you are passing a non-exportable object called options(future.globals.onReference = "error")to detect this mistake sooner, i.e. without having to wait for the call to finish. Then at least you won't have to wait. Either way, note that the brms package already has built-in support for parallelization via futures, so you might not have to use futures yourself. See for example the example code of PS. Please use code-block formatting when pasting code and session out. See https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks for how to do that. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Henrik, Thank you very much for your prompt reply. Also, many thanks for your guidance on the formatting - I am relatively new on GitHub, but I have consulted the guidelines now and will try to stick to them from now on. I realise it can be difficult to give advice without running the exact code and exploring the error yourself. Unfortunately, I cannot share the actual data/model due to confidentiality. They are also quite big, and the model takes several days to run (~4). That was, in fact, my primary motivation for attempting to use the future package - to speed up the 10-fold cross-validation (CV). Being aware that the CV process may take significant time and fail at different stages, I consulted the After reading your response, I tried to rerun the example model/code from In short, with the example model, Importantly, CV runtime was much shorter after loading future and setting Subsequently, I tried to perform the CV after loading future and setting In summary, this "exercise" did not help me understand why the CV of my model failed, as described in my original post. Using the example model from Thanks very much for your help! Best wishes, Tom |
Beta Was this translation helpful? Give feedback.
-
|
Hi, thanks for this. I was under the impression you were rolling your own I've look into brms, and rstan used internally, a little bit more, and, as you also concluded, it is designed to work with futures. Setting So, it is very unlikely that futures are the problem in your case. Instead, I suspect you are running out of memory causing the parallel worker to crash, especially since you're saying these models are very large. Note that if you need X GiB of RAM when you run sequentially, when you run in parallel, each worker adds another X GiB of RAM in requirements. So, if you can barely run it sequentially with X GiB of RAM, you'll need 10*X GiB of RAM if you use 10 parallel workers. I recommend that you try with two parallel workers and see if that works. If that works, try with three workers and so on. You can also try to use OS tools to monitor the memory consumption while this is running. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Henrik, Many thanks for your helpful reply and for documenting the false-positive errors. Regarding the suspected memory-related cause of the crash, please, could I ask you for one more piece of advice? The failed attempt to cross-validate (CV) my model was run on a server (Linux-based; When I tried to run the CV for the very first time, I received this error right at the start: Googling the error, I found this post (https://stackoverflow.com/questions/40536067/how-to-adjust-future-global-maxsize-in-r) that discussed the use of Based on the values reported in the above error message, I set (probably incorrectly) the option as follows: Based on the total system memory (~500 GiB), it looks like there could be enough memory to run the 10-fold CV. Could you please advise me on how to set the Thank you again for your help and advice. Best wishes, Tom |
Beta Was this translation helpful? Give feedback.
-
|
The option huge <- matrix(rnorm(1e9), nrow = 1e5, ncol = 1e4)
f <- future(sum(huge[,1]))This will export the full 100,000-by-10,000 matrix huge <- matrix(rnorm(1e9), nrow = 1e5, ncol = 1e4)
huge_tmp <- huge[,1]
f <- future(sum(huge_tmp))is more efficient. Now, if there is no such improvement the developer can make, then we have at least got their attention and then they can actively raise the limit by setting option The limit only applies per parallel worker, so you do not have to adjust it when you change the number of parallel workers. That you might run out of memory on a parallel worker has nothing to do with this limit. You can run out of memory even when there are no globals exported, e.g. f <- future(too_large <- matrix(rnorm(1e24), nrow = 1e12, ncol=1e12))The error message: only reports on "non-exportable reference" and "total size of globals" as hints that might help us troubleshoot the problem further. These are only hints, or rather suggestions. It is impossible to tell from them if they are the cause for the failure, but I found them very useful to get a first idea. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Henrik, Thanks very much for the clarification. I will try the suggestions given above and hopefully succeed with the CV. Thank you again for all your help. Best wishes, Tom |
Beta Was this translation helpful? Give feedback.
-
Dear all,
Thank you very much for the future package.
Unfortunately, I am experiencing the following issue when using the package.
I estimated a brms::brm() model (Bayesian GAMM for a binary outcome). Subsequently, I have attempted to perform a 10-fold CV for the model using the brms::kfold() function and the future’s multisession plan.
After several days of runtime, (I believe) close to the end of the CV process, the process stopped with the following error message:
Please, can you advise what this error means and how to resolve or avoid it? Many thanks.
This is the code I used to perform the CV that resulted in the error:
Unfortunately, I cannot share the data and/or model due to confidentiality issues. Please note that the data is quite large (n = 80,000) and the model quite complex (includes several spline terms and random effects) - I am not sure if that is relevant to the experienced issue.
Please let me know if you need any additional info.
Thank you very much for your help!
Best wishes,
Tom
PS1 - My system and session info:
PS2 - The full output from R Console:
Beta Was this translation helpful? Give feedback.
All reactions