Not all packages loaded

[mayer79]

Describe the bug
When using foreach on Windows, not all packages are automatically loaded.

Reproduce example

Two examples: one with "ranger", one with "mgcv":

library(ranger)
library(doFuture)
library(foreach)
library(doRNG)

# Set up parallel backend
registerDoFuture()
plan(multisession, workers = 2) 

# Package "ranger" seems not loaded
fit <- ranger(Sepal.Width ~ ., data = iris)
predict(fit, iris[1, ])$predictions

foreach(i = 1:10) %dorng% predict(fit, iris[i, ])$predictions

# Error in { : 
#     task 1 failed - "no applicable method for 'predict' on object of class "ranger""

# Works
foreach(i = 1:10, .packages = "ranger") %dorng% predict(fit, iris[i, ])$predictions

# Now, this works as well
foreach(i = 1:10) %dorng% predict(fit, iris[i, ])$predictions

# Package "mgcv" seems not loaded
library(mgcv)
fit <- gam(Sepal.Width ~ Sepal.Length, data = iris)
predict(fit, iris[1, ])

foreach(i = 1:10) %dorng% predict(fit, iris[i, ])

# Error in { : 
# task 1 failed - "lm object has no valid 'qr' component.

# Works
foreach(i = 1:10, .packages = "mgcv") %dorng% predict(fit, iris[i, ])

# No mistake anymore...
foreach(i = 1:10) %dorng% predict(fit, iris[i, ])

Expected behavior

I would have hoped that using "doFuture" would avoid passing arguments like .packages to foreach().

Session information
Please share your session information, e.g.

R version 4.2.0 (2022-04-22 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=German_Switzerland.utf8  LC_CTYPE=German_Switzerland.utf8   
[3] LC_MONETARY=German_Switzerland.utf8 LC_NUMERIC=C                       
[5] LC_TIME=German_Switzerland.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mgcv_1.8-40     nlme_3.1-157    doRNG_1.8.2     rngtools_1.5.2  doFuture_0.12.2
[6] future_1.28.0   foreach_1.5.2   ranger_0.14.1  

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.8.3      lattice_0.20-45   codetools_0.2-18  listenv_0.8.0     digest_0.6.29    
 [6] parallelly_1.32.1 grid_4.2.0        Matrix_1.4-1      splines_4.2.0     iterators_1.0.14 
[11] tools_4.2.0       parallel_4.2.0    compiler_4.2.0    globals_0.16.1 

[HenrikBengtsson]

(I migrated your original issue to the Future Discussions, because, others might find it useful too, and it is a question relevant to all the futureverse framework - not just to doFuture.)

Hi, yes, in this case you need to guide future here by specifying foreach(..., .packages = "ranger"). This is because, currently, S3 methods for generic function predict() cannot be automatically identify dependent packages. It will know that stats::predict() is needed, but it cannot infer the S3 predict() methods and the ranger package as a dependency.

You'll find another example of this problem in Section 'Missing packages (false negatives)' in the 'Common Issues with Solutions' vignette.

There are additional discussions and ideas on how to improve on this in futureverse/globals#46 and futureverse/globals#47.

[mayer79]

Okay, I though it could have something to do with stats::predict generic, thanks for clarification. The current workaround in the new, superfast "kernelshap" package (https://github.com/mayer79/kernelshap) is to just let (Windows) users specify a named list of foreach arguments.

[HenrikBengtsson]

A poor man's workaround would be to load all packages that are loaded in the main R session also on the worker, e.g.

library(ranger)
library(doFuture)
library(foreach)
library(doRNG)

# Set up parallel backend
registerDoFuture()
plan(multisession, workers = 2)

fit <- ranger(Sepal.Width ~ ., data = iris)


ns <- loadedNamespaces()
y <- foreach(i = 1:10) %dorng% {
  ## WORKAROUND:
  ## Load all packages in worker already loaded in the main session
  dummy <- lapply(ns, FUN = loadNamespace)

  predict(fit, iris[i, ])$predictions
}

It's not the most efficient way, but it might meet your needs.

[mayer79]

Thx for that one!