Package 'future' is not installed on worker [when .libPaths() contains backslashes] #566

HenrikBengtsson · 2021-12-14T04:06:16Z

HenrikBengtsson
Dec 14, 2021
Maintainer

Move from #532 (comment) to here, where @michalovadek asked the following:

I'm running into this error as well when executing plan(multisession) […]: Error: Initialization of plan() failed, because the test future used for validation failed. The reason was: Package 'future' is not installed on worker (r_version: 4.1.1 (2021-08-10); platform: x86_64-w64-mingw32 (64-bit); os: Windows 10 x64 build 19042

I have tried the workaround described here, as well as updating all relevant packages, including to devel versions. The error remains the same throughout.

R version 4.1.1 (2021-08-10) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale: [1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 LC_MONETARY=English_United Kingdom.1252 [4] LC_NUMERIC=C LC_TIME=English_United Kingdom.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] future_1.23.0-9004

loaded via a namespace (and not attached): [1] compiler_4.1.1 parallelly_1.29.0 parallel_4.1.1 tools_4.1.1 listenv_0.8.0 codetools_0.2-18 [7] digest_0.6.29 globals_0.14.0

and:

.libPaths()
[1] "\home.ser.ver.com/home-XO$/user/Documents/R/win-library/4.1"
[2] "C:/Program Files/R/R-4.1.1/library"

Next steps

There is something that causes your parallel workers not to see your .libPaths()[1] where future lives. I don't know, but it could be because it lives on a Windows Network drive (\\home.ser.ver.com) and something is not working as it should when setting up the workers. To troubleshoot this, let's forget about future for now and try with only parallelly.

What does the following output for you:

library(parallelly)
cl <- makeClusterPSOCK(1L, dryrun = TRUE)

and then:

library(parallel)
library(parallelly)
cl <- makeClusterPSOCK(1L)
print(.libPaths())
print(clusterEvalQ(cl, .libPaths()))

Answered by HenrikBengtsson

Dec 14, 2021

Can you try with parallelly 1.29.0-9004;

remotes::install_github("HenrikBengtsson/parallelly", ref="develop")

? I think it fixes it so that you can do:

library(future)
plan(multisession, workers = 2L)

f <- future(42)
v <- value(f)
print(v)

as expected.

View full answer

michalovadek · 2021-12-14T04:41:04Z

michalovadek
Dec 14, 2021

the first output:
Manually, start worker #1 on local machine ‘localhost’ with:

"C:/PROGRA~1/R/R-41~1.1/bin/x64/Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "options(socketOptions = \"no-delay\")" -e "workRSOCK <- tryCatch(parallel:::.workRSOCK, error=function(e) parallel:::.slaveRSOCK); workRSOCK()" MASTER=localhost PORT=11556 OUT=/dev/null TIMEOUT=2592000 XDR=FALSE SETUPTIMEOUT=120 SETUPSTRATEGY=sequential

the second output is identical for both print calls:

[1] "\\\\home.ser.ver.com/home-XO$/user/Documents/R/win-library/4.1"
[2] "C:/Program Files/R/R-4.1.1/library"

0 replies

HenrikBengtsson · 2021-12-14T04:51:06Z

HenrikBengtsson
Dec 14, 2021
Maintainer Author

Good. So, I would expect the following to actually work; please confirm:

library(parallel)
library(parallelly)
cl <- makeClusterPSOCK(1L)
print(clusterEvalQ(cl, library("future")))

Next, let's verify that also the following works:

library(future)
cl <- parallelly::makeClusterPSOCK(1L)
plan(cluster, workers = cl)
f <- future(library("future"))
v <- value(f)
print(v)

If this works, what is the output if you do:

library(future)
plan(multisession, workers = 2L, dryrun=TRUE)

It will produce an error;

Error: MultisessionFuture (future-plan-test) failed to call grmall() on cluster NULL #1 (). The reason reported was 'no applicable method for 'sendData' applied to an object of class "NULL"'

but I'm interested in the lines before that. It'll contain info on something like ... -e '.libPaths(c(...))' ..., which reveals how it attempts to (but obviously fails) set up your .libPaths().

PS. Please format your code blocks here in GitHub using Markdown code blocks to make it easier to read. I did that for your most recent comment.

1 reply

michalovadek Dec 14, 2021

The first two chunks work, I suppose, although the second chunk induced this warning:

Warning message: In for (i in seq_len(n)) { : closing unused connection 4 (<-wl-210629-002.ser.ver.com:11652)

The third chunk gives the following output:

Manually, start worker #1 on local machine ‘localhost’ with:

  "C:/PROGRA~1/R/R-41~1.1/bin/x64/Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "options(socketOptions = \"no-delay\")" -e ".libPaths(c(\"\\home.ser.ver.com/home-XO$/user/Documents/R/win-library/4.1\",\"C:/Program Files/R/R-4.1.1/library\"))" -e "workRSOCK <- tryCatch(parallel:::.workRSOCK, error=function(e) parallel:::.slaveRSOCK); workRSOCK()" MASTER=localhost PORT=11474 OUT=/dev/null TIMEOUT=2592000 XDR=FALSE SETUPTIMEOUT=120 SETUPSTRATEGY=sequential

Manually, start worker #2 on local machine ‘localhost’ with:

  "C:/PROGRA~1/R/R-41~1.1/bin/x64/Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "options(socketOptions = \"no-delay\")" -e ".libPaths(c(\"\\home.ser.ver.com/home-XO$/user/Documents/R/win-library/4.1\",\"C:/Program Files/R/R-4.1.1/library\"))" -e "workRSOCK <- tryCatch(parallel:::.workRSOCK, error=function(e) parallel:::.slaveRSOCK); workRSOCK()" MASTER=localhost PORT=11474 OUT=/dev/null TIMEOUT=2592000 XDR=FALSE SETUPTIMEOUT=120 SETUPSTRATEGY=sequential

Error: MultisessionFuture (future-plan-test) failed to call grmall() on cluster NULL #1 (). The reason reported was ‘no applicable method for 'sendData' applied to an object of class "NULL"

HenrikBengtsson · 2021-12-14T05:27:55Z

HenrikBengtsson
Dec 14, 2021
Maintainer Author

Good. Yes, that warning is expected and can be ignored.

I suspect the problem is with the:

-e ".libPaths(c(\"\\home.ser.ver.com/home-XO$/user/Documents/R/win-library/4.1\",\"C:/Program Files/R/R-4.1.1/library\"))"

part, which I think should be

-e ".libPaths(c(\"\\\\home.ser.ver.com/...\"))"`

That is, some of those backslashes are dropped. I'll have to look into this and I'd imagine this has to be fixed in parallelly.

For now, I think the following is a workaround that should work for you:

library(future)
plan(multisession, workers = 2L, rscript_libs = NULL)

f <- future(42)
v <- value(f)
print(v)

Does that work?

4 replies

michalovadek Dec 14, 2021

it does! thanks a lot

HenrikBengtsson Dec 14, 2021
Maintainer Author

Great. Thanks for confirming. Then I have something to work with. Just one more test for me to confirm my hypothesis. Does the following also work?

library(future)
plan(multisession, workers = 2L, rscript_libs = gsub("\\\\", "\\\\\\\\", .libPaths())

f <- future(42)
v <- value(f)
print(v)

michalovadek Dec 14, 2021

it does

HenrikBengtsson Dec 14, 2021
Maintainer Author

Perfect. So, to conclude, this is a "oversight" bug in parallelly that I'm now tracking in futureverse/parallelly#76.

HenrikBengtsson · 2021-12-14T06:28:49Z

HenrikBengtsson
Dec 14, 2021
Maintainer Author

Can you try with parallelly 1.29.0-9004;

remotes::install_github("HenrikBengtsson/parallelly", ref="develop")

? I think it fixes it so that you can do:

library(future)
plan(multisession, workers = 2L)

f <- future(42)
v <- value(f)
print(v)

as expected.

2 replies

HenrikBengtsson Dec 14, 2021
Maintainer Author

Not sure you get a notification for this one because it was I that created this thread, so posting again with @michalovadek

michalovadek Dec 14, 2021

yep, works, thanks for the quick fix

HenrikBengtsson · 2021-12-17T01:05:25Z

HenrikBengtsson
Dec 17, 2021
Maintainer Author

FYI, parallelly 1.30.0, which fixes this bug, is now on CRAN.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package 'future' is not installed on worker [when .libPaths() contains backslashes] #566

{{title}}

Replies: 5 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Package 'future' is not installed on worker [when .libPaths() contains backslashes] #566

HenrikBengtsson Dec 14, 2021 Maintainer

Next steps

Replies: 5 comments · 7 replies

michalovadek Dec 14, 2021

HenrikBengtsson Dec 14, 2021 Maintainer Author

michalovadek Dec 14, 2021

HenrikBengtsson Dec 14, 2021 Maintainer Author

michalovadek Dec 14, 2021

HenrikBengtsson Dec 14, 2021 Maintainer Author

michalovadek Dec 14, 2021

HenrikBengtsson Dec 14, 2021 Maintainer Author

HenrikBengtsson Dec 14, 2021 Maintainer Author

HenrikBengtsson Dec 14, 2021 Maintainer Author

michalovadek Dec 14, 2021

HenrikBengtsson Dec 17, 2021 Maintainer Author

HenrikBengtsson
Dec 14, 2021
Maintainer

Replies: 5 comments 7 replies

michalovadek
Dec 14, 2021

HenrikBengtsson
Dec 14, 2021
Maintainer Author

HenrikBengtsson
Dec 14, 2021
Maintainer Author

HenrikBengtsson Dec 14, 2021
Maintainer Author

HenrikBengtsson Dec 14, 2021
Maintainer Author

HenrikBengtsson
Dec 14, 2021
Maintainer Author

HenrikBengtsson Dec 14, 2021
Maintainer Author

HenrikBengtsson
Dec 17, 2021
Maintainer Author